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PREFACE 

This  work  partially  fulfills  requirements  of  IDA  Task  T-D5-304,  Ada  Validation, 
for  support  to  the  Ada  Joint  Program  Office  through  analysis  of  technical  or  management 
issues  within  the  Ada  compiler  validation  process. 
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1.0  PURPOSE 


The  purpose  of  IDA  Memorandum  Report  M-376,  Ada  Validation  Workload,  is  to 
provide  the  Ada  Joint  Program  Office  (AJPO)  with  an  analysis  of  the  validation  workload 
for  Ada  compilers  and  the  problems  surrounding  the  annual  expiration  of  an  ACVC 
version.  This  report  is  intended  for  those  individuals  in  the  AJPO  who  develop  the  policy 
and  procedures  that  govern  the  validation  process. 


2.0  SCOPE 


This  paper  focuses  on  the  Ada  Validation  Facilities  (AVFs)  workload  associated 
with  validations  under  Ada  Compiler  Validation  Capability  (ACVC)  version  1.8.  A 
projection  of  workload  growth  has  been  excluded  from  this  analysis.  Section  3  presents 
the  background  and  date  supporting  the  assertion  that  there  is  a  serious  imbalance  in  the 
Ada  Validation  Facility  workload.  Section  4  discusses  findings  concerning  this  imbalance 
and  Section  5  provides  recommendations  for  changes  to  the  Validation  Procedures  and 
Guidelines  and  to  the  AVF  workforce  that  will  improve  the  management  of  that  workload 


3.0  BACKGROUND 


Issuance  of  the  AJPO  Validation  Procedures  and  Guidelines  (Version  1.1)  indicates 
that  the  process  of  validating  Ada  compilers  has  become  a  relatively  stable  set  of 
procedures.  These  procedures  have  undergone  continual  refinements  since  the  inception  of 
validation.  However,  as  much  as  usage  and  analysis  have  identified  and  eliminated  past 
problems  with  the  process,  one  long-standing  problem  remains.  This  problem  is  the 
concentration  of  most  of  the  validation  work  in  the  three  months  surrounding  the  annual 
expiration  of  the  ACVC  version  on  31  May.  The  second  calendar  quarter  (April,  May,  and 
June)  constitutes  a  critical  period  of  intense  demand  on  the  resources  of  the  Certification 
Body.  This  concentration  of  demand  strains  the  process's  human  resources  and  aggravates 
its  incident  problems. 
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The  demand  for  validation  and  re-validation  services  has  increased  significantly 
with  each  ACVC  version  since  version  1.6.  The  total  number  of  validations  completed 
during  the  life  of  each  ACVC  version  is  shown  in  Figure  1.  Even  though  the  number  of 
AVFs  has  also  increased  from  one  to  five,  the  greatest  demand  for  validation  and 
revalidation  services  is  not  evenly  distributable  among  these  five  AVFs. 

Figure  1  Validations  Per  ACVC  Version 

1.1  1.2  1.3  1.4  1.5  1.6  1.7  1.8 

3  0  3  8  7  22  41  122 

Of  the  122  validations  completed  using  ACVC  1.8,  about  two  thirds  of  the  validations  were 
conducted  after  1  April  by  three  AVFs.  Nearly  three  quarters  of  all  the  validation  work  was 
done  from  March  through  June  (see  Figure  2). 

Figure  2  Validation  Demand  for  ACVC  1.8 
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Why  is  there  a  critical  period  of  heavy  validation  demand  surrounding  the  expiration  of  an 
ACVC  version?  What  can  be  done  to  re-distribute  or  manage  this  demand?  This  document 
addresses  these  questions,  analyzes  the  surrounding  issues,  and  proposes  several 
solutions. 


4.0  FINDINGS,  DISCUSSIONS  AND  CONCLUSIONS 


In  Section  4.0,  we  present  the  findings,  discussion,  and  conclusions  concerning  the 
factors  that  contribute  to  the  critical  demand  period,  the  probable  limit  on  human  resources 
that  can  be  available  during  this  period,  and  procedural  changes  that  might  motivate 
vendors  to  schedule  validation  earlier  in  an  ACVC  version  life.  Pertinent  pages  from  the 
Ada  Validation  Procedures  and  Guidelines  have  been  included  in  Appendix  A. 

4.1  VENDOR  BEHAVIOR 

4.1.1  Finding 

During  ACVC  1.8,  vendor  behavior  indicates  that  there  is  a  tendency  to  validate  or 
re- validate  as  late  as  possible  in  the  ACVC  version  life.  Sections  4.1  and  4.2  are  being 
interpreted  as  official  permission  to  complete  validation  with  an  expired  ACVC  version  if 
vendors  can  show  the  AVF  that  they  have  used  that  version  for  pre-validation  testing  before 
it  expired  (31  May).  (See  Appendix  A  for  the  specific  parts  of  Sections  4.1  and  4.2  that 
address  scheduling,  use  of  an  expired  ACVC,  and  automatic  extension  of  certificates.) 

4.1.2  Discussion 

The  Procedures  and  Guidelines  Section  4.1  is  being  interpreted  to  mean  that  a 
compiler  vendor  may  be  permitted  to  validate  a  compiler  using  an  expired  version  of  the 
ACVC  if  the  vendor  has  submitted  pre- validation  materials  to  the  AVF  before  1  June.  This 
interpretation  ignores  the  requirement  that  "the  AVF  has  had  sufficient  time  to  determine 
that  there  are  no  failures."1  AVF  experience  indicates  that,  on  the  average,  at  least  two 


1  Ada  Compiler  Validation  Procedures  and  Guidelines,  Version  1.1, 1  January  1987,  p.  10,  Section  4.1.4. 
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weeks  should  be  allowed  for  analysis,  resolution  of  issues,  and  preparation  of  the  draft 
validation  summary  report  (VSR)  for  Ada  Joint  Program  Office/Ada  Validation 
Organization  (AJPO/AVO)  review  and  comment.  The  number  of  validations  completed 
after  31  May  (with  an  expired  test  suite)  indicates  that  several  AVFs  imposed  the  31  May 
deadline  for  submission  of  pre-validation  materials  but  performed  analysis  of  these 
materials  later. 

Section  4.2  requires  that  a  vendor  submit  a  compiler  for  revalidation  in  a  timely 
manner  (as  defined  in  Section  4.1,  Step  4,  page  10),  in  order  to  qualify  for  automatic 
extension  of  the  validation  certificate  (VC).  Thus  the  Procedures  and  Guidelines  require 
that  the  vendor  give  consideration  to  the  time  estimates  given  in  Section  4. 1  for  completing 
the  validation  process,  including  issuing  a  certificate.  However,  some  vendors  wishing  to 
re-validate  have  not  reflected  this  consideration  of  the  time  required  to  analyze  pre¬ 
validation  materials  and  to  complete  the  subsequent  steps  leading  to  a  certificate. 

Of  the  43  re-validations  under  ACVC  1.8, 18  were  done  after  the  expiration  of  the 
VC.  Of  the  24  revalidations  that  took  place  one  month  or  more  prior  to  the  expiration  of  a 
VC,  9  of  these  were  early  re-validation  under  ACVC  version  1.8  rather  than  version  1.9 
with  its  increased  coverage.  Another  5  revalidations  were  performed  just  prior  to  the  end 
of  the  3-month  extension  period  that  all  1.6  VCs  received.  Only  10  of  these  24 
revalidations  were  performed  significantly  in  advance  of  the  compiler's  VC. 

In  the  pre-April  period,  the  validation  workload  was  evenly  divided  between  new 
validations  and  revalidations.  In  the  critical  period,  revalidations  numbered  about  the  same 
as  before;  however,  the  bulk  of  the  demand  during  the  last  quarter  was  for  testing 
compilers  that  had  not  been  validated  before.  When  vendors  encountered  development 
problems,  it  was  necessary  to  re-schedule  validation  even  later.  There  was  some  indication 
that  even  experienced  vendors  planned  first-time  validations  as  late  as  possible. 

4.1.3  Conclusions 

•  The  Procedures  and  Guidelines  document  is  being  interpreted  in  a  way  that 
permits  informal  extensions  of  the  12-month  ACVC  version  life. 

•  The  vendor  demand  pattern  evident  during  ACVC  1.8  (and  1.7)  will  persist  in 
subsequent  ACVC  cycles.  Since  there  is  no  incentive  for  early  validation  and 
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re- validation,  testing  under  AC  VC  1.9  will  be  concentrated  in  the  last  quarter 
and  continue  through  July,  or  later. 

4.2  ALTERNATIVE  PRACTICES 

4.2.1  Finding 

The  AJPO  should  consider  feasible  approaches  for  reducing  vendor  tendencies  to 
schedule  late  and  for  reducing  peak  AVF  workloads  caused  by  the  expiration  of  an  ACVC 
version. 

4.2.2  Discussion 

We  have  found  that  three  approaches  are  feasible.  They  are: 

•  extending  certificate  life; 

•  providing  for  late  validation;  and, 

•  fixing  certificate  expiration  dates. 

4.2.2. 1  Extended  Certificate  Life 

The  extended  certificate  life  approach  attempts  to  use  exhibited  vendor  behavior  to 
more  evenly  distribute  the  demand  for  services.  Since  new  vendors  can  be  expected  to 
validate  as  late  as  possible  in  the  ACVC  version  life,  we  can  leave  this  period  open  for  new 
vendors  by  issuing  a  certificate  that  expires  later  than  12  months  from  the  date  of  issue 
(e.g.,  16  months).  The  central  idea  is  to  issue  a  certificate  that  has  a  longer  life  than  the 
ACVC  version  so  that  certificates  issued  at  one  point  in  the  ACVC  version  cycle  will  expire 
at  a  different  point.  Although  the  present  year’s  large  number  of  critical-period  validations 
would  still  present  a  concentrated  demand  for  revalidation,  this  demand  would  come  at  a 
different  point  in  the  ACVC  cycle  from  the  expected  heavy  demand  for  new  validations. 
The  peaks  in  demand  for  re-validation  may  begin  to  even  out  as  some  vendors  rely  on 
automatic  extensions  of  their  certificates  to  do  re-validations  at  a  later  date.  However,  the 
concentration  of  re-validation  workload  is  not  likely  to  be  significantly  dispersed 
throughout  the  year. 
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4. 2. 2. 2  Limited  Provision  for  Late  Validation 

This  alternative  provides  conditions  for  using  an  ACVC  test  suite  version  after  its 
expiration  date.  It  is  intended  to  remove  a  particular  date  from  being  the  focus  of  last- 
minute  validation  efforts  by  allowing  additional  time  for  vendors  to  complete  their 
implementation  while  imposing  a  reduction  in  the  life  of  the  validation  certificate  that  can  be 
received  as  a  result  of  using  an  expired  versions  of  the  ACVC  for  validation. 

In  this  alternative,  disputed  tests  must  be  submitted  by  the  vendor  before  expiration 
of  the  ACVC  version  that  will  be  used  for  validation.  Pre-validation  materials  may  be 
submitted  to  the  AVF  after  the  expiration  date  of  the  ACVC  version,  but  these  materials 
must  reflect  the  AVO  resolution  of  any  disputed  tests.  On-site  testing  should  begin  within  a 
month  following  submission  of  the  pre-validation  materials.  If  on-site  testing  completes 
successfully,  the  vendor  will  be  awarded  a  validation  certificate  with  an  expiration  date  that 
is  equal  to  the  test  suite  version's  expiration  date  plus  one  year.  It  is  expected  that  this 
provision  for  late  validation  will  not  be  used  repeatedly  by  a  vendor.  The  AVF  can 
discourage  vendors  from  habitual  late  validation  through  contract  negotiations. 

The  certification  body  (AJPO,  AVO,  AVFs)  may  define  other  conditions  to  be  met 
in  order  to  qualify  for  late  validation.  For  example,  it  may  be  desirable  to  establish  a  cut¬ 
off  date  for  submission  of  a  letter  of  intent,  or  for  submission  of  pre-validation  materials, 
or  for  completing  on-site  testing.  These  deadlines  may  be  desirable  so  that  an  AVF  can 
arrange  to  preserve  its  capability  to  use  the  expired  ACVC  without  impact  on  validations 
with  the  current  ACVC.  The  late  validation  alternative  is  limited  by  the  requirement  for 
submission  of  disputed  tests  before  the  ACVC  version  expires  and  may  include  other 
deadlines  that  can  be  imposed  by  the  certification  body. 

4. 2. 2. 3  Fixed  12-Month  Certificate 

The  fixed  12-month  certificate  approach  addresses  the  vendor  tendency  to  schedule 
validation  near  or  after  the  expiration  date  of  an  existing  certificate.  With  this  alternative, 
the  expiration  date  of  a  certificate  is  always  12  months  from  the  expiration  date  of  the 
previous  certificate  (rather  than  12  months  from  the  date  of  issue  as  is  the  current  practice). 
Vendors  who  validate  earlier  than  the  expiration  date  do  not  lose  some  period  of  validated 
status  under  a  current  certificate.  Vendors  who  wait  to  schedule  until  just  before  or  after  a 
certificate  expiration  date  do  not  gain  the  additional  time  granted  by  an  automatic  extension 
because  the  new  certificate  expiration  date  will  be  fixed  at  12  months  from  the  expiration 
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date  of  the  old  certificate.  Thus  vendors  with  current  validation  certificates  would  be  less 
motivated  to  schedule  later  and  might  start  validation  earlier.  This  motivation  can  be 
reinforced  by  requiring  the  vendor  to  schedule  or  begin  on-site  testing  prior  to  the 
expiration  of  the  current  certificate  in  order  to  qualify  for  an  automatic  extension. 

4.2.3  Conclusions 

The  Extended  Certificate  Life  alternative  will  help  segregate  workloads  for  re¬ 
validations  from  validation  of  new  implementations,  but  does  not  directly  address  vendor 
tendency  to  schedule  late  in  an  ACVC  version  life  or  after  expiration  of  a  VC.  Limited 
Provision  for  Late  Validation  and  the  Fixed  12-month  Certificate  alternatives,  taken 
together,  provide  motivation  for  vendors  to  cooperate  with  AVFs  in  managing  workload 
schedules.  Thus  vendor  motivation  is  in  the  form  of  disincentives  for  delays  in  scheduling 
and  completing  validation. 

4.3  AVF  WORKLOAD 

4.3.1  Finding 

The  three  AVFs  with  the  heaviest  workloads  are  those  that  have,  or  soon  will  have 
contracts  in  place,  for  validation  services  of  Ada  compilers.  Only  one  of  these  AVFs  can 
augment  the  workforce  during  the  critical  period. 

4.3.2  Discussion 

Contractor  support  for  validation  services  is  intended  to  provide  personnel  who  are 
skilled  in  analytical  and  on-site  testing  procedures  for  one  or  more  programming  languages. 
These  contracts  provide  an  AVF  with  a  labor  pool  but  there  is  some  limit  as  to  the  numbers 
of  people  with  experience  in  Ada  validation  who  can  be  available  during  the  peak  workload 
period  for  Ada  validation.  Figure  3  is  an  estimate  of  the  number  of  contractor  personnel 
that  can  be  used  by  AVFs  to  perform  technical  tasks  associated  with  Ada  compiler 
validation. 
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Figure  3  Estimate  ot  Contractor  Personnel 

AVF  No.  of  Contractors 

Wright-Patterson  Air  Force  Base  16 

National  Bureau  of  Standards  (NBS)  6  (projected) 

Ministry  of  Defense,  UK  JL(projected) 

30 

Wright-Patterson  uses  these  contractor  personnel  in  teams  consisting  of  two  people 
per  validation  projects  with  each  team  is  capable  of  completing  one  project  per  month.  At 
NBS  and  in  the  UK,  teams  are  generally  not  used.  The  analyst  hours  per  validation  project 
can  vary  from  150  to  350  hours;  however,  on  the  average,  the  output  for  NBS  and  the  UK 
is  one  validation  project  per  person-month  (14  per  month).  Thus  the  combined  validation 
service  capacity  for  these  three  AVFs  is  about  22  validation  projects  per  month.  Wright- 
Patterson  is  the  only  AVF  that  can  increase  the  number  of  available  teams  if  there  is 
adequate  lead-time  to  locate  or  train  analysts.  However,  this  expansion  of  resources  has 
some  limit,  as  yet  undefined,  in  terms  of  available  funds  and  human  resources. 

The  demand  for  last  quarter  validation  under  ACVC  1.8  at  these  three  AVFs  was  82 
validations.  Even  though  this  number  includes  several  validations  that  were  completed 
within  a  single  project  (requiring  only  one  on-site  visit  by  the  AVF),  the  demand  for  ACVC 
1.9  validations  and  re- validations  will  likely  exceed  the  supply  of  AVF  resources  as  new 
customers  are  added  to  the  demand  pattern.  The  extra  workforce  that  Wright-Patterson 
may  be  able  to  obtain  during  the  critical  period  will  be  a  crucial  factor  in  being  able  to 
accommodate  the  workload  for  ACVC  1.9.  The  establishment  of  an  additional  AVF  in  the 
U.S.,  with  a  contract  modeled  on  the  current  Wright-Patterson  contracting  vehicle,  would 
provide  more  flexibility  not  only  for  peak  workload  periods  but  also  for  meeting  an  overall 
increase  in  workload. 

4.3.3  Conclusion 

The  capability  of  the  AVFs  in  the  US  and  in  the  UK  to  accommodate  an  expanding 
workload  for  Ada  validation  services  is  limited.  Concentration  of  demand  during  the 
critical  period  in  1988  will  expose  these  limitations. 
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5 . 0  RECOMMENDATIONS 

5.1  LIMITED  PROVISION  FOR  LATE  VALIDATION 

Permit  a  limited  use  of  an  expired  ACVC  version  by  incorporating  a  Limited 
Provision  for  Late  Validation  in  the  wording  of  the  Ada  Validation  Procedures  and 
Guidelines,  paragraph  4.1.  The  following  revised  wording  is  proposed:  (the  new  wording 
has  been  italicized) 

Paragraph  4.1  STEPS  AND  PROCEDURES.  There  are  nine  steps  in  the 
Ada  VALIDATION  process  that  a  VENDOR  and  the  CERTIFICATION 
BODY  must  successfully  complete.  It  is  expected  that  the  first  seven  of 
these  steps  will  be  completed  prior  to  the  expiration  of  the  ACVC  version 
that  is  used  in  VALIDATION  testing.  These  steps  are: 

Paragraphs  4.1.1  through  4.1.3  -  no  change. 

Paragraphs  4.1.4  -  Last  paragraph  within  4.1.4  (ref.  page  10,  Ada 
Validation  Procedures  and  Guidelines). 

In  the  event  that  a  VENDOR  completes  step  three  in  the  VALIDATION 
process  (i.e.,  Negotiate  a  formal  agreement  and  submit  disputed  ACVC 
tests)  prior  to  the  expiration  date  of  the  current  ACVC,  and  that  the 
expiration  date  is  subsequently  reached  before  the  VALIDATION  process  is 
complete,  then  the  VENDOR  is  allowed  to  proceed  with  the  original  ACVC 
provided  that: 

a.  The  AVF  determines  that  there  are  no  failures  of  APPLICABLE 
ACVC  TESTS,  and 

b .  the  VENDOR  does  not  submit  test  disputes  after  the  expiration  of 
the  ACVC  version  that  will  be  used  for  validation,  and 

c.  the  AVF  agrees  to  complete  the  VALIDATION  process  after 
giving  consideration  to  already  scheduled  work. 
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Completing  the  VALIDATION  after  the  expiration  of  an  ACVC  version 
is  a  late  VALIDATION.  It  is  intended  that  there  will  be  a  limited  use  of  the 
provision  for  late  VALIDATION  by  VENDORS.  The  AVF  may  impose 
additional  conditions  that  must  be  met  by  the  VENDOR  to  ensure  that  a 
VENDOR  does  not  habitually  rely  on  late  VALIDATION. 

5.2  FIXED  12-MONTH  CERTIFICATE 

Provide  explicit  qualifying  conditions  for  an  automatic  extension  of  a 
VALIDATION  CERTIFICATE  by  incorporating  a  Fixed  12-Month  Certificate  in  the 
wording  of  Section  4.2.  The  following  wording  is  offered  for  consideration.  Changes  to 
the  current  wording  have  been  italicized. 

Paragraph  4.2  CERTIFICATE  EXPIRATION.  VALIDATION 
CERTIFICATES  expire  in  accordance  with  the  time  period  established  by 
the  Director,  AJPO.  This  time  period  is  currently  one  year  after  the  date  of 
issue  when  the  VENDOR  has  completed  step  7  of  the  VALIDATION 
process  prior  to  expiration  of  the  ACVC  version  used  for  VALIDATION. 
Subsequent  VALIDATION  CERTIFICATES  issued  for  this  BASE 
COMPILER  will  have  an  expiration  date  that  is  fixed  at  one  time  period 
beyond  that  of  the  previous  VALIDATION  CERTIFICATE  when  the 
VENDOR  has  completed  step  7  of  the  VALIDATION  process  prior  to  the 
expiration  date  of  the  ACVC  version  used  for  VALIDATION.  When  a 
VENDOR  has  completed  step  7  of  the  VALIDATION  process  after  the 
expiration  date  of  the  ACVC  version  used  for  VALIDATION,  the  expiration 
date  of  the  VALIDATION  CERTIFICATE  will  be  one  time  period  beyond 
the  expiration  date  of  the  ACVC. 

A  BASE  COMPILER  and  associated  DERIVED  COMPILERS  will  be 
removed  from  the  list  of  VALIDATED  Ada  COMPILERS  when  the 
VALIDATION  CERTIFICATE  for  the  BASE  COMPILER  expires.  The 
VENDOR  may  request  that  the  AJPO  not  remove  the  BASE  COMPILER 
and  associated  DERIVED  COMPILERS  from  the  list  of  VALIDATED  Ada 
COMPILERS  when  the  VENDOR  has: 
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a.  begun  step  7  of  the  VALIDATION  process  prior  to  the  expiration 
date  of  the  VALIDATION  CERTIFICATE  for  the  BASE  COMPILER, 
or 

b .  requested  an  AVF  to  schedule  step  7  prior  to  the  expiration  date  of 
the  VALIDATION  CERTIFICATE  for  the  BASE  COMPILER  but  has 
been  delayed  because  the  AVF  had  prior  commitments  for 
VAUDATION. 

It  is  the  intent  of  the  CERTIFICATION  BODY  that  VENDORS  schedule  the 
REVALIDATION  of  their  VALIDATED  COMPILERS  to  occur  in  advance 
of  the  expiration  of  the  associated  VALIDATION  CERTIFICATE.  By 
applying  a  rule  for  establishing  the  expiration  date  at  a  fixed  time  period  that 
is  relative  to  the  previous  expiration  date,  a  VAUDATION  CERTIFICATE 
extends  the  time  that  an  Ada  COMPILER  is  a  VAUDATED  COMPILER  by 
a  fixed  amount,  regardless  of  when  the  actual  testing  occurs.  Thus,  all 
VENDORS  may  complete  step  7  well  before  the  expiration  date  of  a 
VAUDATION  CERTIFICATE  without  concern  for  redundant  coverage." 

5.3  AVF  WORKLOAD 

A  decision  to  establish  an  additional  AVF  should  be  based  upon  analysis  of  the 
workload  for  ACVC  1.9.  This  analysis  should  also  include  the  geographic  distribution  of 
clients  and  criteria  for  selecting  an  organizational  entity  that  will  be  chartered  as  an  AVF. 
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